Computers and mobile devices (e.g., smart phones, tablet computers, handheld computers, and the like) are increasingly often equipped with input devices such as cameras that display and/or record streaming video of the physical world surrounding the operator and the device. The video images or streaming video may be thought of as representations of the “real world” or “reality.” The technology of augmented reality (AR) combines elements of such representations of the physical world with digital or virtual objects. The final augmented vision or display gives the appearance to the user that the digital object exists or is present in the real world or surrounding physical environment.
For AR to be enjoyable, the user experience should be seamless. For example, a preferred AR experience may include having the virtual object presented in a display screen or on a monitor so as to appear to be correctly positioned within the real or physical world, and the AR experience may then include enabling user interaction with the correctly positioned object on their computer or mobile device (also called a “viewing device” herein).
Furthermore, the AR graphics should adjust in real time in response to changes in the user's real world position or movement of the viewing device (or its camera). This requires accurate registrations of the user's viewing device and the digital data providing the streaming video of the physical world and also providing the virtual or AR object.
Accurate registration of world coordinates to AR coordinates for virtual objects is a major problem that has not been adequately addressed in the field of AR. The lack of precision results in digital models drifting across the user's device as the background or physical world images change, in AR objects jumping out of position, and/or in AR objects or digital models appearing in an unexpected locale or position in the displayed real world. Within the AR field, there is also a need to quickly and accurately update the AR graphics in response to a user's change in position (or, more accurately, a change in the position and/or orientation of the viewing device) to yield an augmented display output providing an accurate and seamless AR experience.
Varied approaches have been tried in the AR industry to try to address these problems, with two approaches dominating: location-based AR through the use of a Global Positioning System (GPS) and the use of targets or markers (e.g., high-contrast objects). In location-based AR on a mobile device, the on-board GPS provides a location that is correlated with the digital data being viewed in AR. Problems are encountered because the GPS location provided by the mobile device is often inaccurate and not of the precision necessary for precise viewing in a real time AR system. Many mobile devices lack true GPS, and coordinates are provided by triangulation from Wi-Fi and cellular towers.
A second approach to address the problem of inaccurate location in AR systems is the use of high-contrast markers. High-contrast markers or objects generally have regions that can be easily distinguished by a computer system in colors such as black and white, red and green, or some other color combination of colors that can be easily distinguished from each other by a computer system. In such an AR system, markers are introduced into the physical environment to aid the hardware and software operating on mobile devices or mobile viewing devices in performing object tracking and registration.
Marker technology includes use of fiducial markers that have been encoded with data, with better known technologies including use of bar codes (one dimensional (1D) or two dimensional (2D)) or quick response (QR) code technology. Information is encoded into these markers, such as for commercial tracking by the United States Postal Service and the like, and decoded by light scanners (e.g., for 1D barcodes) or charge-coupled devices (e.g., for 2D barcodes and QR codes). Further, markers in AR have been used to convey a position of the marker in relation to the camera or viewing device. In many of these approaches, a digital image or AR object is associated with the marker and, in the resulting AR system, a virtual object is projected on or near the marker. While each of these approaches provides some advances or advantages, the AR experience is not as seamless as desired with problems in using markers remaining in a typical AR system (e.g., a viewing device with AR software or the like).
The foregoing examples of the related art and limitations related therewith are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of skill in the art upon a reading of the specification and a study of the drawings.